Developing a Test Collection to Support Diversity Analysis

Promoting diversity in search has gained much recent interest, however there exists a lack of sufficient benchmarks for evaluation. In this paper, we describe the creation of a new test collection for evaluating diversity in image search. To ensure the creation of a realistic resource, query variations from an image search log were used as the basis for identifying topics that might require a search system to produce diverse output. We describe the development of our benchmark being used in the ImageCLEFPhoto 2009 image retrieval task.